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DETAILED ACTION 



1. This action is responsive to communications: amendment filed 2/5/2004, amendment 
filed on 10/27/2003, application filed on 03/23/2000. 

2. Claims 13-26 and claim 41 have been cancelled. 

3. Claims 1-12 and 27-40 are pending in the case. Claim 1, 27, and 29 are independents 
claim. 



The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 



4. Claims 1-2, 6, 8-11, 27-28, 29-30, 34, and 36-39 are rejected under 35 U.S.C. 103(a) 
as being unpatentable over Ho et al. (hereafter referred to as Ho), US 6,021,412 filed 
4/2/1996 in view of Shiiyama et al. (hereafter referred to as Shiiyama), US 6,247,009 Bl 
filed 3/9/1998 and Rosenbaum, US 5,404,435 patented 4/4/1995. 

Regarding independent claims 1, 27, and 29, Ho teaches extracting one or more 
document keywords from a document in col. 1 lines 50-55. Ho also teaches collecting one or 
more images associated with the document in col. 1 line 55 - col. 2 line 5, but Ho does not teach 
that the images have a location in the document. Although Ho teaches matching concept 
keywords to images, Ho does not teach is generating a proximity factor for each image and each 
document keyword that reflects the degree of correlation between the image and the document 



Claim Rejections - 35 USC § 103 
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keyword. Although Ho teaches determining the importance of each image to a document, Ho 
does not teach determining the importance of each of the plurality of images according to an 
image metric that combines the proximity factors for each document keyword and image pair. 

Shiiyama teaches an image query method which includes comparing a keyword or 
keywords to an image and generating a correlation, which Shiiyama calls a matching rate, 
between the image and the keyword or keywords in fig. 4, col. 3 lines 65-67, col. 5 line 6 - col. 6 
line 36, col. 7 lines 40-58, col. 8 line 65 - col. 9 line 1, and col. 10 line 24 - col. 1 1 line 19. 
Shiiyama teaches combining proximity factors of multiple keywords in order to determine the 
importance of an image in fig. 4, col. 5 line 6 - col. 6 line 36, col. 8 line 65 - col. 9 line 1, and 
col. 10 line 24 - col. 1 1 line 19. The matching rates of a plurality of keyword and image pairs 
are sorted to present the user with the most desirable images. 

Rosenbaum teaches collecting non-text objects, such as image objects, from a document 
in the abstract and col. 1 line 67 - col. 2 line 2. It would have been obvious to one of ordinary 
skill in the art at the time the invention was made to have combined Rosenbaum into Ho to have 
created the claimed invention. It would have been obvious to have collected the images from a 
document to make the images more readily available. It would have further been obvious to one 
of ordinary skill in the art at the time of the invention to have determined the importance of the 
plurality of images collected from the document using the keyword identification taught by Ho to 
create a proximity factor for each of the plurality of images as is taught by Shiiyama. Using the 
sorted results based on matching factor taught by Shiiyama, it would have been obvious to one of 
ordinary skill to have enabled the presentation of the images collected from the document to the 
user in order of importance based on the correlations of the keyword and images. 
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Regarding dependent claims 2 and 30, Ho teaches presenting images determined to be 
important on a display device in col. 6 lines 18-20. Ho does not teach that these images are 
located in the document, however Rosenbaum teaches collecting non-text objects, such as image 
objects, from a document in the abstract and col. 1 line 67 - col 2 line 2. It would have been 
obvious to one of ordinary skill in the art at the time the invention was made to have combined 
Rosenbaum into Ho to have created the claimed invention. It would have been obvious and 
desirable to have collected the images from a document make the images available so that their 
importance could have been determined. 

Regarding dependent claims 6 and 34, Ho teaches identifying image text associated 
with each image of the plurality of images in Fig. 5 item 506 and Fig. 6 and col. 1 lines 47-50. 
Ho does not teach that these images are located in the document, however Rosenbaum teaches 
collecting non-text objects, such as image objects, from a document in the abstract and col. 1 line 
67 - col. 2 line 2. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have combined Rosenbaum into Ho to have created the claimed 
invention. It would have been obvious and desirable to have collected the images from a 
document make the images available so that their importance could have been determined. 

Regarding dependent claims 8 and 36, Ho teaches searching the metadata information 
associated with the image for text describing the image in Fig. 5 item 506 and col. 1 lines 47-50. 
Ho does not teach that these images are located in the document, however Rosenbaum teaches 
collecting non-text objects, such as image objects, from a document in the abstract and col. 1 line 
67 - col. 2 line 2. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have combined Rosenbaum into Ho to have created the claimed 
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invention. It would have been obvious and desirable to have collected the images from a 
document make the images available so that their importance could have been determined. 

Regarding depend claims 9 and 37, Ho does not explicitly teach that the metadata 
information is compatible with hypertext markup language. One of ordinary skill in the art at the 
time of the invention would have known how to make metadata compatible with hypertext 
markup language. It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to combine the their knowledge with Ho to have created the claimed 
invention. It would have been obvious and desirable to use hypertext markup language 
compatible metadata because hypertext markup language is the most common form of storing 
metadata and it would have been advantageous to use it to create an invention compatible with 
well known standards. 

Regarding dependent claims 10 and 38, Ho teaches matching concept words to images, 
but does not explicitly teach lexically analyzing the image text associated with the image and the 
keyword to determine the degree of correlation between the image and the keyword. Shiiyama 
does teach lexically analyzing the image text associated with the image and the keyword to 
determine the degree of correlation between the image and the keyword in fig. 4, col. 3 lines 65- 
67, col. 5 line 6 - col. 6 line 36, col. 7 lines 40-58, col. 8 line 65 - col. 9 line 1, and col. 10 line 
24 -col. 11 line 19. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to have combined Shiiyama into Ho to have created the claimed invention. It would 
have been obvious and desirable to determine the degree of correlation between the keywords 
and the images to best determine the most important images in the document, just as an image 
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search query determines the degree of correlation between search keywords and images to 
present a user with the most desirable images in a database. 

Regarding dependent claims 11 and 39, Ho teaches matching concept words to images, 
but does not explicitly teach performing a phonetic comparison between the image text 
associated with the image and the document keyword to determine the degree of correlation 
between the image and the document keyword. Shiiyama does teach performing a phonetic 
comparison between the image text associated with the image and the document keyword to 
determine the degree of correlation between the image and the document keyword in fig. 4, col. 
3 lines 65-67, col. 5 line 6 - col. 6 line 36, col. 7 lines 40-58, col. 8 line 65 - col. 9 line 1, and 
col. 10 line 24 - col. 1 1 line 19. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to have combined Shiiyama into Ho to have created the claimed invention. It would 
have been obvious and desirable to determine the degree of correlation between the keywords 
and the images to best determine the most important images in the document, just as an image 
search query determines the degree of correlation between search keywords and images to 
present a user with the most desirable images in a database. 

Regarding dependent claim 28, Ho teaches presenting images determined to be 
important on a display device in col. 6 lines 18-20. Ho does not teach that these images are 
located in the document, however Rosenbaum teaches collecting non-text objects, such as image 
objects, from a document in the abstract and col. 1 line 67 - col. 2 line 2. It would have been 
obvious to one of ordinary skill in the art at the time the invention was made to have combined 
Rosenbaum into Ho to have created the claimed invention. It would have been obvious and 
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desirable to have collected the images from a document make the images available so that their 
importance could have been determined. 

5. Claims 3-5 and 31-33 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Ho et al. (hereafter referred to as Ho), US 6,021,412 filed 4/2/1996 in view of Shiiyama 
et al. (hereafter referred to as Shiiyama), US 6,247,009 Bl filed 3/9/1998 and Rosenbaum, 
US 5,404,435 patented 4/4/1995 as applied to claims 1 and 29 above, and further in view of 
Dutta, US 6,480,837 Bl filed 12/16/1999. 

Regarding dependent claims 3 and 31, Ho does not teach ordering the document 
keywords according to an ordering criterion or weighting the proximity factor associated with 
each document keyword and image pair based on the order of the document keyword. Dutta 
does teach ordering the document keywords according to an ordering criterion and weighting the 
proximity factor associated with each document keyword and image pair based on the order of 
the document keyword in Fig. 2. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Dutta into Ho in view of Shiiyama and Rosenbaum to have created the 
claimed invention. It would have been obvious and desirable to order and weight the keywords 
according to a popularity weight naturally the keyword-image pairs of the more popular 
keywords were more popular than the keyword-image pairs of the less popular keywords. 

Regarding dependent claims 4 and 32, Ho does not teach that the frequency that each 
document keyword appears in the document determines the ordering criterion used to order the 
document keywords. Dutta does teach that the frequency that each document keyword appears 
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in the document determines the ordering criterion used to order the document keywords in col. 1 
lines 23-37. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Dutta into Ho in view of Shiiyama and Rosenbaum to have created the 
claimed invention. It would have been obvious and desirable to order and weight the keywords 
according to a popularity weight naturally the keyword-image pairs of the more popular 
keywords were more popular than the keyword-image pairs of the less popular keywords. 

Regarding dependent claims 5 and 33, Ho does not teach that the ordering criterion 
orders the document keywords according to their relationship with the subject matter of the 
document. Dutta does teach that the ordering criterion orders the document keywords according 
to their relationship with the subject matter of the document in col. 4 lines 25-42. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Dutta into Ho in view of Shiiyama and Rosenbaum to have created the 
claimed invention. It would have been obvious and desirable to order and weight the keywords 
according to a popularity weight naturally the keyword-image pairs of the more popular 
keywords were more popular than the keyword-image pairs of the less popular keywords. 

6. Claims 7 and 35 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ho 
et aL (hereafter referred to as Ho), US 6,021,412 filed 04/02/1996 in view of Shiiyama et aL 
(hereafter referred to as Shiiyama), US 6,247,009 Bl filed 3/9/1998 and Rosenbaum, US 
5,404,435 patented 04/04/1995 as applied to claims 6 and 34 above, and further in view of 
Matsumoto, US 6,526,170 filed 12/13/1994. 
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Regarding dependent claims 7 and 35, Ho does not teach scanning a bit-mapped 
representation of the image for text information or converting the bit-mapped representation of 
the text information into image text. Matsumoto does teach scanning a bit-mapped 
representation of the image for text information and converting the bit-mapped representation of 
the text information into image text in Fig. 7 and col. 1 lines 23-25. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to combine Matsumoto into Ho in view of Shiiyama and Rosenbaum to have created 
the claimed invention. It would have been obvious and desirable to extract text from the image 
itself and use that to match with the keywords so that the image-keyword match would further 
increase in accuracy. 

7. Claims 12 and 40 are rejected under 35 U.S.C. 103(a) as being unpatentable over Ho 
et al. (hereafter referred to as Ho), US 6,021,412 filed 04/02/1996 in view of Shiiyama et al. 
(hereafter referred to as Shiiyama), US 6,247,009 Bl filed 3/9/1998 and Rosenbaum, US 
5,404,435 patented 04/04/1995 as applied to claims 1 and 29 above, and further in view of 
Cullen et al. (hereafter referred to as Cullen), US 6,397,213 Bl filed 5/12/1999. 

Regarding dependent claims 12 and 40, Ho does not teach identifying the location of 
the image in the document, measuring the distance in the document between the image and the 
document keyword, or determining the correlation between the document keyword and the 
image according to the distance between the document keyword and the image. Cullen does 
teach determining the distance between an image in a document and a keyword in the document 
in fig. 4, 8C, col. 2 line 66 - col. 3 line 3, and col. 9 lines 8-19. The distance is determined 
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manually by a user in Cullen through repeating a series of mouse clicks to discover the distance 
between the keyword and the image. However, this could have easily been automated by one of 
ordinary skill in the art since the coordinates of both the image and the keywords are known. 

It would have been obvious to one of ordinary skill in the art at the time of the invention 
to have combined Cullen into Ho in view of Shiiyama and Rosenbaum to have created the 
claimed invention. It would have been obvious and desirable for one of ordinary skill in the art 
at the time of the invention to have used to the image-keyword distance determination taught by 
Cullen to have modified the calculation of the correlation for the keyword-image pair taught by 
Shiiyama so that the keywords nearby an image, and probably related to the image would have 
had higher importance than the images far away from the keywords in the document. This 
would have generally caused the image metric to determine that important images within the 
document are located near the highest concentrations of keywords in the document. Cullen 
demonstrates the importance of this as the mouse clicking action expanding the keyword search 
radius is used to determine how close and how many keywords are located near the target image. 
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Response to Arguments 

8. Applicant's arguments with respect to claims 1-13 and 27-40 have been considered but 
are moot in view of the new ground(s) of rejection. The Examiner has applied Shiiyama in place 
of Sciammarella in the rejection of claims 1-2, 6, 8-11, 27-28, 29-30, 34, and 36-39. Shiiyama 
teaches correlating one or more keywords to an image and sorting the images for presentation to 
a user based on the respective correlations of each keyword-image pair. The Examiner has 
additionally introduced Cullen in the rejection of claims 12 and 40. Cullen teaches determining a 
distance between an image contained in a document and document keywords. The coordinates 
within the document of both the image and the keywords are known by Cullen. 

Conclusion 

9. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. Takata et al., US 6,526,400 Bl filed 9/30/1999 discloses an image search apparatus 
which acquires associative words in relation to an input query word, and makes a keyword 
search of image information on the basis of the obtained associated words and input query word. 
Barber et al., US 5,751,286 patented 5/12/1998 discloses an image query system. 

10. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Peter J Smith whose telephone number is 703-305-593 1 . The 
examiner can normally be reached on Mondays-Fridays 7:00am-3 :30pm. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Joseph H Feild can be reached on 703-305-9792. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



PJS 

May 13,2004 
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